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E valuate! The command is so familiar in the 
prevention literature that a program without 
an evaluation component is likely to be 
starved for support. Indeed, the U.S. Department of 
Education’s periodic Grant Competition to Prevent 
High-Risk Drinking or Violent Behavior Among 
College Students requires applicants to develop an 
evaluation plan for their proposed projects. Part 
86 — the Drug-Free Schools and Campuses 
Regulations— :^uires that, every two years, campuses 
determipg'^eff^^ of their alcohol and other 
dm^ffre™tion programs 
^nd TOlement needed 
ch»ges. In addition to 
valuating their own 
^ efforts, campuses are 
encouraged to imj 
evalualfidrjfo^ams. 

Given that b^^^ound 
it is summing to find that 
evaluation of 
alcohol and other drug 
preyentionstra 
gies is neglected on many 
campuses. 

Klking to veterans in the prevention field sheds 
^me light on what’s wrong. Prevention workers may 
^g their evaluation responsibility because 
they’re expteiijg too much too soon. For example, it 
[campus culture of high-risk 
drinking in a''STW!i|||rifl^ but all too often 
that is an expectation of mose^wlfe^pport preven- 
ion efforts directed at students. 

Dude, who has been working in prevention 
,on thP^niversity of Missouri campus for more 
£^rs, is impatient with the focus on 

rates in measuring the effect of 
prevention. ‘T>toe that phrase, but it seems to be 
the yardstick. Everything is in the 'binge-drinking’ 
basket, and if that doesn’t change, then forget it.” 



The Drug-Free Schools 
and Campuses 
Regulations require 
that . . . campuses 
determine the 
effectiveness of their 
alcohol and other drug 
prevention programs . . . 




Dude is an ardent supporter of environmental 
strategies and publicizing social norms as a means 
of changing student behavior but she is equally 
devoted to education as part of the picture. She 
emphasizes that immediate results may not be meas- 
urable by a convenient gauge such as rates of “binge 
drinking” turned up in student surveys. 

“My feeling is that we need to get our students to 
progress through the range of changes in behavior. 
They start with Tm thinking about my behavior.’ 
Then, Tm thinking about my friend’s behavior.’ We 
want to move them down 
that continuum. However, 
the behavior change may 
not happen when they’re in 
college.” 

William Dejong, Ph.D., 
from his vantage point as 
the former director of the 
U.S. Department of 
Education’s Higher 
Education Center for Alcohol 
and Other Drug Abuse and 
Violence Prevention and a 

professor of social and behavioral sciences at the Boston 
University School of Public Health, sees a widespread 
failure by colleges and universities in compiling the 
information needed to design a meaningful evaluation 
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of their prevention work. Dejong led an evalua- 
tion of social norms marketing involving 32 
different institutions (l6 intervention sites and 
l6 control sites) and found that none of them 
had any archival data they could use to meas- 
ure changes in problems and behaviors on 
their campuses. 

“Did their student health centers have intake 
forms asking whether the presenting problem 
was alcohol related? No. Did their housing 
departments keep track of how much money 
they’re spending on vandalism repair? No. Did 
their campus police forms indicate consistently 
whether a student apprehended or arrested had 
been drinking? No. On a typical campus, each 
department was too wrapped up in its own 



Principles of 
Effectiveness for 
Prevention Programs 

The U.S. Department of Education’s Office 
of Safe and Drug-Free Schools promotes 
principles of effectiveness for prevention pro- 
grams, as codified in the No Child Left 
Behind Act of 2001. A subset of the princi- 
ples of effectiveness that are most applicable 
to institutions of higher education can be 
summed up as follows: 

• Design programs based on a thorough 
needs assessment of objective data. 

• Establish a set of measurable goals and 
objectives linked to identified needs. 

• Implement prevention activities that 
research or evaluation have shown to be 
effective in preventing high-risk drinking 
or violent behavior. 

• Use evaluation results to refine, improve, 
and strengthen the program and refine 
goals and objectives as appropriate. ■ 



affairs to think about alcohol problems in 
terms of the campus as a whole,” he said. 

Dejong concedes that deciding to do better 
data collection is only step one. 

“Once you have record-keeping forms in 
place you have to train people to use them. 
You have to monitor them to make sure 
they’re consistently filling out the forms cor- 
rectly. It’s time intensive and despite its value 
a lot of our campuses just don’t go there. But 
without good data it’s very difficult to really 
gauge whether you’re making progress.” 

The U.S. Department of Education’s Office 
of Safe and Drug-Free Schools (OSDFS) 
places a strong emphasis on program evalu- 
ation. Deborah A. Price, assistant deputy sec- 
retary of OSDFS, points to the principles of 
effectiveness for prevention programs prom- 
ulgated in 1998 (see sidebar). While these 
principles focus primarily on K-I2, they also 
are applicable to higher education. 

“These principles call for a deliberate 
process for program development that rests 
on sound evaluation processes — ^from data 
collection and assessment to goal setting and 
measuring whether goals and objectives have 
been met,” Price says. “Following these prin- 
ciples helps us understand what works and 
what doesn’t so that we can improve the 
effects of prevention programs.” 

In fact, the OSDFS Alcohol and Other 
Drug Prevention Models on College 
Campuses grant competition seeks to iden- 
tify evaluated programs with proven effec- 
tiveness for dissemination to the field. 

According to William Modzeleski, associ- 
ate assistant deputy secretary in OSDFS, the 
grant competition’s goal is to move the 
field toward more effective practice. 

“Programs selected through the grant 
competition — and there have been 34 since 
1999 — all included rigorous evaluation in 
their program planning and implementation. 
That was key in their ability to provide 



evidence that the initiative was effective in 
reducing alcohol- and other drug-related 
problems,” said Modzeleski. 

Not Students Alone 

While student surveys are a mainstay of efforts 
to measure how students are responding to pre- 
vention efforts, this kind of evaluation suffers 
from a degree of skepticism. Are students hon- 
est and accurate in what they say in surveys 
about drinking? A recent study at Hobart and 
William Smith Colleges indicates that such 
self-reporting may be more accurate than 
many critics believe. This study also demon- 
strates how departments not usually involved 
in prevention can make a significant contribu- 
tion to the effort. 

David W. Craig, Ph.D., is a chemist and the 
director of the Alcohol Education Project at 
Hobart and William Smith Colleges. His depart- 
ment has been drawn into a demonstration of 
how breathalyzer tests can be used to determine 
whether the drinking behavior reported by stu- 
dents when they return to their dorms after a 
night of partying is consistent with their actual 
blood alcohol levels. Craig and his team took 
breathalyzer samples from 1,837 students in a 
study designed carefully for randomization and 
other considerations to ensure accuracy and 
safety. The results bolstered the validity of self- 
reports as a measure of how much students 
are drinking. 

“What we see is that there is a substantial 
error for given individuals. Some people overre- 
port. Some people underreport,” Craig explains. 
“So we have high values and low values, but if 
you collect a large enough sample they cancel 
out. As long as you have a large enough sam- 
ple of self-reporting, you’re going to get a good 
representation of the actual distribution of blood 
alcohol concentrations and risk levels, based on 
our research using breathalyzers.” 

Craig acknowledges that using breathalyzer 
data as an evaluation tool is in its infancy, but 
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he is impressed by the willingness of students ! 
to participate in his study No fewer than 85 j 

percent of the 2, l6l students went along with | 

the breath test when they were assured of | 
anonymity. i 

“If you look at a mail-in survey you’ll never : 
in the world see a response rate of 85 percent. j 
You’re lucky to get 50 percent. So this partici- I 

pation rate exceeds anything I’ve ever seen in : 

terms of survey work. ’ ’ i 

Craig points out that his breathalyzer study j 

cost nothing at all. “It was integrated into our j 

curriculum. All of the research scientists who ! 
did the data collection were independent-study i 
research students getting course credit to do | 
the research.” | 

Missouri’s Kim Dude also relies on help and i 

advice from other departments to come up with ! 
valid designs for evaluating a prevention strategy, j 
Still, an evaluation may require money ranging I 
from pay for research assistants to incentives that : 
lure students into responding to a survey. | 

“But it’s an investment,” she says. “We’re j 

more likely to get other funding if an evalua- I 
tion shows we’ve been effective. The evaluation i 
not only helps us retain grants but also i 

increases the likelihood of getting more grants | 
or even more university money. We evaluate vir- | 
tually everything we do, and for a lot of reasons. : 
The most important reason is that we want to i 
get better all the time.” j 

In addition, the cost of evaluation may be I 
more than offset by discontinuing programs : 
and activities that are found to be ineffective. | 
Ideally, the effect of a campus-based preven- i 
tion effort would be compared with what is I 
happening on a similar campus that was not i 
the target of such an effort. Such comparison- : 
based studies can be challenging to design and | 
carry out but can have substantial effect as an | 
evaluation of a prevention strategy. One example : 
is a trial reported in the \0um2lAcididi0n (Vol. i 
100, No. 3, March 2005) to test the efficacy of a j 
campaign to reduce drinking and driving : 



Programs 

among students at a public university in the 
Southwest. Another university with a student 
body of similar characteristics served as a 
comparison site. 

The study, designed and carried out by John 
D. Clapp, Ph.D., and colleagues at San Diego 
State University and the Pacific Institute for 
Research and Evaluation, provided for stepped- 
up driving under the influence (DUI) enforce- 
ment, a media advocacy campaign, and a 
social marketing campaign on one campus 
while only routine DUI enforcement continued 
at the comparison university. Using baseline 
data collected in advance, the researchers 
measured a considerable drop in self-reported 
driving after drinking among students at the 
university where the prevention effort had 
taken place. 

Such comparison studies in the realm of 
research lie well beyond the means of most 
campus prevention programs, but they illustrate a 
point made by Dejong. If an evaluation indicates 
that a prevention effort had a disappointing out- 
come, a good question to ask is: compared with 
what? Doing nothing at all? That’s why many 
studies include intervention sites and control sites, 
where data are collected on the same measures 
during the same time period. 

“There might be a school that is actually 
seeing an increase in problem drinking on its 
campus in spite of all the work prevention peo- 
ple are doing,” Dejong says. “Does that mean 
their prevention program is failing? If they look 
at data from comparison schools they might find 
that the problems on their own campus are not 
going up as much as they are at other schools. 
Being able to gauge how you’re progressing 
against other schools is important, too.” 

Editor's note: Tips on designing prevention pro- 
grams with effective evalmtion components 
can be found on the Web site of the National 
Institute on Alcohol Abuse and Alcoholism at 
http://wu)w. collegedrinkinff)revention.gov . ■ 



Message From 
Deborah Price, Assistant 
Deputy Secretary for Safe 
and Drug-Free ^hools 

With greater emphasis 
on accountability when 
it comes to funding, 
evaluation becomes 
even more important. 

But it is not just those 
providing funding for 
prevention programs — government agencies, 
foundations, colleges and universities — ^that ben- 
efit from good evaluations that show whether pro- 
grams are being effective. Programs themselves 
need to engage in ongoing evaluations so that 
they initiate changes midstream to achieve greater 
effectiveness. 

The Office of Safe and Drug-Free Schools 
supports program evaluation in a number of 
ways. First, our Higher Education Center for 
Alcohol and Other Drug Abuse and Violence 
Prevention provides technical assistance to 
campuses and communities on ways to con- 
duct evaluation through a number of publica- 
tions, training sessions, and workshops 
(see http://www.higheredcenter.org/evaD . 

Our annual National Meeting on Alcohol 
and Other Drug Abuse and Violence Prevention 
in Higher Education has included pre-confer- 
ence institutes on evaluation as well as work- 
shops showcasing the evaluation experiences 
on specific campuses. In addition, we showcase 
the work of the model program awardees, 
selected on the basis of evaluation demonstrating 
the effectiveness of their programs. Our new pub- 
lication Experiences in Effective Prevention: 

The US. Department of Education's Alcohol 
and Other Drug Prevention Models on 
College Campuses Grants highlights the eval- 
uation lessons learned by 22 grantee institu- 
tions funded from 1999 to 2004 by the U.S. 
Department of Education’s Alcohol and Other 
Drug Prevention Models on College Campuses 
grants initiative. ■ 
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Evaluation on a Shoestring 



I n a perfect world, every prevention pro- 
gram would include enough money for 
rigorous evaluation. We don’t, however, 
live in a perfect world. All too often it seems 
that evaluation is almost an afterthought, 
with most of the resources directed to program 
implementation instead. 

“We don’t have the money to evaluate,” says 
John D. Clapp, Ph.D., professor in the School 
of Social Work and Public Health and director 
of the Center for Alcohol and Other Drug 
Studies and Services at San Diego State 
University as well as a member of the Higher 
Education Center’s Review Group. “It’s pretty 
rare for people without grant money from the 
National Institutes of Health to do a full evalu- 
ation of outcomes.” 

Nevertheless, evaluation is necessary. In fact, 
according to John H. Pryor, director of the 
Cooperative Institutional Research Program at 
the University of California, Los Angeles’s 
(UCLA) Higher Education Research Institute, it 
is “incredibly important.” 

“If you’re going to do a program, you really 
have an obligation to evaluate,” he says. “We 
shouldn’t be doing it on a shoestring. If people 
had been evaluating effectively from the begin- 
ning, we’d be much further along.” 

The imperative to measure a program’s 
effectiveness coupled with the dearth of evalu- 
ation dollars means that prevention practition- 
ers must use their resources wisely. Finding 
inexpensive ways to gather data can help. Web- 
based surveys, because they incur no expenses 
for postage, printing, or materials, have 
become increasingly popular. Web technology 
saves staff time, too. 

Another economical way to collect information 
is to use focus groups rather than individual 
interviews for program feedback. It is less expen- 
sive, for example, to hold a one-hour discussion 
with 12 participants in a focus group than to 
engage in 12 one-hour interviews. 



Some prevention programs lend themselves 
to using data compiled by outside entities, and 
these data cost essentially nothing. A munici- 
pal or university police department, for exam- 
ple, may keep track of the number of college 
students who are detained for alcohol-related 
offenses. A residence hall may tally property 

‘If you’re going to 
do a program, you 
really have an 
obligation to 
evaluate” 

damage. Many colleges regularly survey their 
student populations through institutional 
research offices. If these surveys include ques- 
tions about alcohol or other drugs, prevention 
practitioners can obtain information from 
them. At UCLA, the freshman survey “Your 
First College Year” asks one question about 
high-risk drinking and one about participa- 
tion in prevention programs. Because the sur- 
vey leaves space for miscellaneous questions, 
prevention practitioners can ask that specific 
questions be added. 

“You can reach a large population that 
way,” says Pryor. 

In addition to making changes in the way 
information is gathered, evaluators can trim 
costs by decreasing the frequency of survey 
administration. Many prevention programs 
include several strategies or events, and sur- 
veys for each one are often unnecessary or 
“overkill,” according to Pryor. Instead, pro- 
grammers can conduct their evaluation after 
all of the events have occurred, saving money 
by using one instrument instead of several. 

Gloria T. DiFulvio, Ph.D., research assistant 
professor at the University of Massachusetts 
Amherst School of Public Health and Health 
Sciences, advocates developing partnerships and 



relationships across campuses and communities 
as a way to decrease evaluation expense. Other 
departments may be willing to share costs, 
and a campus and community coalition may 
help with the workload. For two-year cam- 
puses and for campuses without research 
institutions, personnel at nearby campuses 
can provide expertise. 

At larger universities, DiFulvio points out 
that graduate students in need of projects 
might be interested in collecting and analyz- 
ing data about prevention programs. Students 
who are engaged in independent study are 
another source of assistance. 

Pryor agrees that graduate students and 
others can be useful. But he cautions that 
evaluation is not the same as research, and he 
encourages prevention practitioners to find 
people who are trained to do evaluation. 

“Evaluation is an applied setting,” he says. 
“It’s real people in real circumstances. It’s not 
like a psychology experiment in a closed 
room. As an evaluator, you have to be able to 
move with what’s going on with the program. 
In a strict research design, you can’t change.” 

Even with the incorporation of the above 
cost-cutting strategies, measuring long-term 
behavior change — so-called outcome evalua- 
tion — remains expensive. But process evalua- 
tion, which looks at a program as it unfolds 
and measures intermediate change, is less 
costly. Process evaluators ask what should 
happen and when. They decide which indica- 
tors are important for change and set up a 
system to track those indicators. 

Here is how it works. 

A university program to decrease the nega- 
tive consequences of high-risk drinking, for 
example, might feature educational work- 
shops for fraternities and sororities, responsi- 
ble beverage server training, advertising 
campaigns, and other strategies. In a process 
(Continued on page 5) 
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evaluation of the program, the first question 
to ask about each activity is: Did it occur as 
planned? If an advertising campaign called 
for 500 posters and only 50 were displayed, 
the activity did not occur as planned. Then, 
did the activity reach its target audience? A 
workshop intended for fraternities may not 
be as successful if it only attracts athletes or 
community business owners. Next, if an 
intervention is identified as a good one, what 
about it has been successful? As program plan- 
ners answer these and other process questions 
and evaluate intermediate goals by looking at 
specific indicators at various points in time, they 
can make needed corrections. 

Policymakers are 
starting to favor model 
programs when they 
make funding decisions. 

Sometimes even process evaluation is not 
feasible due to limited resources but that does 
not mean program planners cannot both 
gauge the progress of their efforts and save 
money by using a logic model (see p. 11). 
Planners must ask themselves ahead of time 
what attitudes and behaviors would indicate 
change and how these attitudes and behaviors 
could be measured. In a program to reduce 
DUI, for example, prevention practitioners 
could posit that drinking and driving will be 
decreased if students believe that their chance 
of DUI arrest is high. So activities that are 
intended to increase perception of risk would 
be put in place. Then, students would self- 
report DUI and evaluators would ask them 
about their perception of risk. A decrease in 
self-reported DUI and a concurrent increase in 
perceived risk of arrest would lead to the 
assumption that the particular program 



reduced DUI. But under these conditions, quan- 
tification is not possible. 

The use of model prevention programs can 
cut evaluation costs dramatically. These model 
programs are science-based interventions that 
have been tested and proven effective under 
various criteria developed by a number of dif- 
ferent agencies, including the U.S. Department 
of Education, the Center for Substance Abuse 
Prevention, or the National Institute on Alcohol 
Abuse and Alcoholism. They are available to 
universities across the country with instructions 
about what to do and how to do it. The theory 
is that if campuses are faithful to the model 
programs, the long-term outcomes will mirror 
those that were achieved when the programs 
were tested. Thus the evaluation becomes, “Did 
the program get implemented as it should have 
been?” says Clapp, adding that policymakers 
are starting to favor model programs when they 
make funding decisions. 

Statewide initiatives, in which campuses work 
together to share resources and help one other 
with data collection, can help defray the cost of 
evaluation. There are currently 47 statewide ini- 
tiatives for college alcohol abuse prevention, with 
long-standing programs in Illinois, Ohio, Maine, 
New York, and Pennsylvania, among others. 



No matter which cost-saving techniques and 
methods are used, DiFulvio says that evalua- 
tion should be seen as a core component of 
any prevention program. 

‘'Evaluation is not 
just about monitoring 
student outcomes, 
it is about guiding 
implementation. ” 

“When budgeting, we need to understand 
that part of the budget is for evaluation and 
include it from the beginning,” she says. 
“Evaluation is not just about monitoring stu- 
dent outcomes, it is about guiding implemen- 
tation. So it is important to communicate with 
those who may be in supervisory positions and 
who may not recognize the value of evaluation.” 

IPs a difficult task and one that, thus far, 
has demanded creativity and resourcefulness. 

But evaluation “on a shoestring,” although 
hardly optimal, remains an important part of 
prevention programs by providing colleges 
and universities with information on whether 
they have, at minimum, implemented the 
activities that they intended to. ■ 



Office of Safe and Drug-Free Schools 

If you would like more information about the Office of Safe and Drug-Free Schools (OSDFS), 
please visit the office’s Web site at http://www.ed.gov/OSDFS . For more information about the 
office’s higher education initiatives, please contact: 

Richard Lucey, Jr., Education Program Specialist, Richard.Lucey@ed.gov : 202-205-5471 
Ruth Tringo, Education Program Specialist, Ruth.Tringo@ed.gov : 202-260-2838 
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Dealing With Alcohol and Other 
Drug Issues at Small Colleges 

$0 What’s the Problem, Anyway? 

by Laurence W. Mazzeno 



A t a March 2007 conference sponsored 
by the National Association of Student 
Personnel Administrators, I was asked 
to appear on a presidents’ panel with leaders 
of two of America’s distinguished large public 
institutions, the University of Rhode Island 
and Texas Tech University. What an incongru- 
ous pairing, I thought. After all, Alvernia 
College, where I served as president for eight 
years, is radically different from theirs. The 
entire student population is one seventh that 
of URI, and notably smaller than the entering 
freshman class at Texas Tech. Moreover, 
Alvernia’s endowment is roughly equivalent to 
the annual interest earned by endowments at 
many larger, more prestigious centers of higher 
education. If I may be allowed a bit of hyper- 
bole for effect, I think it fair to say that at a 
place like Penn State or the University of 
California my annual budget would have been 
considered a rounding error. 

Generally those of us not in that elite classi- 
fication U.S. News & World Report describes 
as “national liberal arts colleges” have actu- 
ally tried to make a virtue of these differences, 
marketing ourselves as places where students 
receive individual attention from faculty and 
administrators keenly interested in helping 
them succeed in college and after. Try as hard 
as we might, however, we simply can’t seem to 
get over the fact that the behemoths in our 
industry have more money — and keep getting 
more whenever someone identifies a problem. 
We tend to become especially envious when we 



see six-figure grants going to large public 
institutions or to “Ivy League caliber” schools. 
Many of us pine for the chance to show that, 
if we just had money, we too could solve all 
our problems. 

But I would like to offer a “minority opin- 
ion” on this matter when it comes to alcohol 
and other drug prevention and treatment. I 
think many of us would only squander more 
money if it were made available. We’re simply 
not ready to use it effectively. 

Why? A snapshot of my college can help 
answer that question. Founded by the 
Bernardine Franciscan Sisters in 1958, Alvernia 
enrolls slightly fewer than 3,000 students in 
more than 30 undergraduate and graduate 
programs. Half the students are “nontradi- 
tional” (over the age of 25). We have taken 
special pride in touting our mission as a 




Alvernia President Emeritus Laurence W. Mazzeno 



values-centered institution where religious 
principles guide all our actions. We highlight 
the fact that our faculty are teachers first, and 
that every student gets personal attention. We 
are proud of the comprehensive program run 
by our student affairs professionals that pro- 
motes healthy lifestyles. 

We are not unique, of course; there are 
hundreds of similar institutions throughout 
America that claim strong religious and 
moral underpinnings and insist that they are 
in touch with their students. The sad fact is 
that in many cases these attitudes and beliefs 
are actually hurting rather than helping col- 
leges deal with drug and alcohol problems 
on campus. 

I was reminded of that recently when I was 
helping recruit new member institutions for 
the Network Addressing Collegiate Alcohol and 
Other Drug Issues (Network). I sent letters to 
presidents of nearly 800 small and medium- 
sized colleges to explain what services the 
Network could offer. I didn’t receive many 
responses, but among the replies one from the 
president of a small religiously affiliated insti- 
tution much like mine caused me to do a 
mental double-take: “Dear Dr. Mazzeno,” this 
president wrote. “Thank you for inviting our 
college to become a member of The Network. 
We appreciate what The Network is doing; 
however, we will not be joining. None of our 
students uses alcohol or drugs.” 

Now it is certainly possible that what this 
president told me is true, but it has been my 

(Continued on page 7) 
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experience that no college in America is 
totally immune from alcohol and other drug 
problems. My surprise was muted, however, 
when I recalled what had happened only 
seven years earlier on my own campus. In the 
spring of 1998, my chief student affairs offi- 
cer, who had come to Alvernia with me in 
1997 , gave her first briefing to trustees about 
the status of alcohol and other drug issues on 
campus. She and I both thought the report 
was fairly innocuous and routine. The 
trustees, however, stared in disbelief, then 
erupted with a volley of outraged responses: 
“We’ve never heard this before!” “We have a 
policy against having alcohol anywhere on 
campus!” “We’re a Catholic school — how can 
there be problems with alcohol and drugs 
here?” Apparently they were convinced that 
strong policies coupled with a tradition of 
religious affiliation were all we needed to 
create a drug-free utopia. In short, they were 
in denial 

Actually there seems to be a lot of this 
denial going on. Leaders at many small col- 
leges are convinced that alcohol and other 
drug problems on their campuses are either 
nonexistent or not significant. Why do they 
think this? Because they believe in the great 
myth that permeates such institutions: 
because we’re small, we know all our students 
personally. This cliche has been a mantra 
behind which faculty and staff and leadership 
at small colleges have hidden for decades. As a 
consequence, many of us have never taken 
any formal steps to learn about those who are 
enrolling at our institutions. On my campus, 
it took participation in the National Survey of 
Student Engagement to convince people we 
were not nearly as well informed as we 
thought — and certainly not always pleased 
with what we learned. That tool and other, 
similar surveys opened our eyes in a number 



of ways. Not only did we discover that our 
freshmen did not really feel connected to their 
fellow students or their faculty as well as we 
assumed — though seniors, on the other hand, 
were very comfortable in the level of their 
engagement — but we learned that our stu- 
dents were remarkably like those on other 
campuses, including large research universi- 
ties: They came to us from a culture that con- 
doned, and even promoted, the use of alcohol 
and other drugs, and they were not about to 
let the fact that they were attending a Catholic 
college change behaviors they’d learned long 
before they enrolled at Alvernia. 

As painful as all this was, once we got over 
our denial and began to rely on fact-based 
information, we began to look for ways to get 
beyond platitudes and really help our students 
deal with alcohol and other drug issues. At 
that point the task became remarkably more 
manageable: We discovered there were many 



resources available, from federal, state, and 
private organizations — much of which came 
at no cost except the time we were willing to 
invest in dealing with alcohol and other drug 
matters honestly and effectively. 

The good news is that now trustees, faculty, 
and staff are willing to admit there’s a prob- 
lem; that is a tiny but important step forward. 
We also found the courage to admit that we 
would probably not eradicate alcohol and 
other drug problems, at least not in the fore- 
seeable future. Nevertheless, leaders who face 
this dreary fact head-on rather than hiding 
from the truth behind the comforting myths 
about life at small colleges will know that 
they are making a positive difference in the 
lives of those who attend their institutions. 
Can we settle for anything less? 

Laurence W. Mazzeno, Ph.D., is president 



emeritus of Alvernia College in Reading, Pa. ■ 
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The U.S. Department of Education’s Higher Education Center for 
Alcohol and Other Dmg Abuse and Violence Prevention has published 
Experiences in Effective Prevention, prepared on behalf of the 
Office of Safe and Dmg-Free Schools. 
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identified model programs at 22 institutions of higher education. This publication discusses the 
seven core elements of the success of those model programs that can be adapted for other campuses. 

To view online or download Experiences in Effective Prevention, please visit the Center's Web site 
(http://www.higheredcenter.org/pubs). You may order a free print or CD version online or by calling 
1-800-676-1730; TDD Relay-friendly Dial 711. 
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It’s Only Logical by Linda Langford 



W hen campus prevention coordina- 
tors implement a program, activity, 
or policy, they are doing so in hope 
of achieving some result, such as decreased high- 
risk drinking or violent behavior among college 
students (the program goal). A logic model is a 
diagram that shows the program planners’ com- 
monsense understanding of how and why pro- 
gram activities lead to program goals. 

Why use a logic model? Because a logic 
model lays out all the steps that are expected to 
occur, it serves as an easy-to-follow program 
plan. Also, the very process of generating a 
logic model serves to create a shared under- 
standing among staff members of the pro- 
gram’s process, goals, and objectives, which helps 
project staff to work together more effectively. 

The logic model also serves as the basis for 
the evaluation plan. Evaluation planners can 
decide how and when to measure each step of 
the program specified in the logic model. Using 
the logic model in this way ensures that each 
important program component is measured. 
Because program staff participated in generat- 
ing the logic model, evaluators are more likely 
to collect information that is relevant to pro- 
gram staff and useful for improving the program. 

In addition, measuring each element in the 
logic model assists in the interpretation of eval- 
uation results. If evaluation results show that 
the program goals were not achieved, program 
staff can ascertain where in the sequence the 
program did not work as planned. If the antici- 
pated outcomes were obtained, having data 
that support each step in the logical sequence 
improves the argument that the long-term 
goals occurred as a result of the program activ- 
ities. In essence, the logic model allows pro- 
gram staff to use evaluation results to report on 
the accuracy of their assumptions about how 
activities would lead to goals. 

But before constructing a logic model, pro- 
gram planners should conduct a needs assess- 
ment designed to gather information on the 
nature of the alcohol, other drug, and violence- 
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related problems in their particular campus 
and community setting. For example, one 
campus might find that many problems take 
place in and around local community bars that 
engage in aggressive promotions targeted at 
students and have lax enforcement of underage 
drinking and public order laws. Another cam- 
pus might find greater alcohol, other drug, and 
violence problems associated with fraternity 
parties or large events. Each of these problems 
suggests the need for a different solution. 

If you are having 
trouble specifying the 
change, ask yourselves, 
“what result(s) do we 
expect from this 
activity?’' 

A needs assessment might include adminis- 
tering a student survey; scanning the environ- 
ment using the Higher Education Center’s 
College Alcohol Risk Assessment Guide: 
Environmental J^proaches to Prevention 
and campus safety audit tools; and reviewing 
data gathered by campus and community 
agencies, such as vandalism reports, alcohol- 
related hospital visits, campus disciplinary 
actions, visits to rape crisis centers, and com- 
munity arrests. Reviewing these data can help 
program planners to identify the specific prob- 
lems that should be addressed. After identifying 
and prioritizing the problems, planners consult 
prior research and theory to determine which 
strategies are likely to be effective in addressing 
these priority problems, and then create a set of 
integrated program activities. 

This is the best time to construct a logic 
model: after program activities have been ten- 
tatively selected but prior to program imple- 
mentation. Constructing a logic model as a 
part of this planning process can serve to make 
explicit how the planned activities will lead to 
the ultimate goals for these efforts (e.g., decreased 
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high-risk drinking, decreased incidents of vio- 
lence). Planning a prevention program is a 
complex undertaking, and even experienced pro- 
fessionals often find that the first draft of a logic 
model reveals flaws in the logical sequence of the 
program. A logic model allows these flaws to be 
corrected on paper before implementing pro- 
gram activities in the field. 

There is no single format for a logic model. 
One way of constructing a logic model is to list 
program activities on the far left-hand side of a 
piece of paper and program goals on the far 
right. The planning team can then fill in the 
middle of the diagram by brainstorming the 
series of steps by which each activity leads to 
the goals. The trickiest part of this process is to 
articulate each interim step as a change rather 
than describing the activity. For example, 
rather than stating the step as “workshops in 
dorms,” specify what will be different as a 
result of that activity, that is, “increase in stu- 
dent knowledge about campus sexual assault 
policies” (or whatever the workshop content 
is). Beginning each link with “increase” or 
“decrease” can help in articulating each step 
as a change. If you are having trouble specify- 
ing the change, ask yourselves, “what result(s) 
do we expect from this activity?” 

It is important to articulate all of the steps 
that you expect will occur. For example, we do 
not expect most training programs to result 
directly in behavior change. Instead, we 
assume there are intermediate steps, such as 
increased knowledge about the topic, increased 
positive attitudes about the behavior change, 
and perhaps an increase in skills needed to per- 
form the new behavior. Each of these assump- 
tions should be made explicit in the model. It 
is especially important to articulate both how 
initial changes in knowledge and attitudes will 
lead to a desired behavior change and how the 
behavior change itself is linked to the ultimate 
program goals. For example, what changes in stu- 
dent behavior are you assuming will result from 
increased knowledge about sexual assault policies, 
(Continued on page 11) 
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Q&A With Peggy Glider 

Peggy Glider, Ph.D., is the coordinator of evaluation and research for the Campus Health Service at the 
University of Arizona. She has served as the principal investigator or research director on many federal 
research and demonstration grants in the alcohol, other drug abuse, and violence arenas. She also has par- 
ticipated in national panels and evaluation teams for the Public Health Service and US Department of 
Education. In addition, she has provided evaluation technical assistance to a variety of colleges and universi- 
ties and serves as an evaluation consultant for the Higher Education Center for Alcohol and Other Drug 
Abuse and Violence Prevention. The University of Arizona is a member of the Network Addressing Collegiate 
Alcohol and Other Drug Issues. 



Q: Often evaluation takes a back seat when 
programs are in the planning stage. What would 
you say to encourage people to build in evalua- 
tion as a key component of prevention efforts? 

A: I tell people that in light of increasing 
emphasis on accountability, without evaluation 
it is very difficult for a program to demonstrate 
any effect on the targeted problems or popula- 
tion. Building evaluation in from the very 
beginning allows for the monitoring of what is 
going on in terms of process as well as out- 
comes. If the program is not getting the desired 
results, it can be tweaked as it goes along. 

Evaluation is also critical in order to get 
funding. Solid evaluation that has gone hand- 
in-hand with the program as it is developed 
provides the program with a much better posi- 
tion for grant writing to request funding. 

Proactive inclusion of evaluation in program 
development also helps in terms of cost effec- 
tiveness. If evaluation is part of the program 
design from the beginning, appropriate infor- 
mation can be collected to figure out what is 
working and what isn’t. For example, if a less 
expensive format of a program is working just 
as well as a more expensive format — say a 
small group versus individual sessions — that 
allows for some tweaking to save money. In 



addition, there is no way to demonstrate effec- 
tiveness without evaluation data. 

Q: What are some of the ways to use evalua- 
tion to help programs meet their goals? 

A: Evaluation can help determine if the target 
audience is being reached. It can provide qual- 
itative information as well as “head counts” of 
who is participating in activities. Evaluation 
can help tailor program outreach efforts and 
what is being provided to whom. But it is also 
critical for determining outcomes. The only 
way to know if a program is working is 
through evaluation efforts. The major goal of 
prevention programs is to see some change in 
knowledge, attitude, and, we hope, behavior. 

Q: What kinds of goals can people realistically 
set and evaluate during relatively short funding 
periods in order to satisfy the evaluation 
requirements or expectations of funders? 

A: I strongly encourage the use of a logic 
model. It helps people to really look at what 
they are doing, why they are doing it, and what 
changes they can expect in the short term, 
which may just be students coming in — that 
important process information. What are some 



of the intermediate things that might be 
expected to happen in three months or six 
months as well as the long-term effect? We 
would all love to see underage drinking or 
high-risk drinking on campuses cut in half, 
but we know that certainly will not happen in 
the short term. Thinking about both the long- 
term outcomes as well as what is reasonable to 
achieve in the short and intermediate terms is 
part of the planning process when it comes to 
determining what some reasonable expecta- 
tions might be. 

Q: Often evaluation goals focus on shifts or 
changes in individual behaviors or individual 
consequences. However, current research, such 
as the NIAAA [National Institute on Alcohol 
Abuse and Alcoholism] panel reports, points to 
the need for environmental management to 
influence the campus and community environ- 
ment. Are there ways that programs can build in 
some ways to evaluate the environment as well? 

A: Absolutely. There are several tools that are 
very good for conducting environmental scans 
available through the Higher Education Center 
as well as other sources. An environmental 
scan, which is a process for developing a cam- 
pus profile by taking a look around and having 

(Continued on page 10) 
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Join the Network! 



Q&A With Peggy Glider 

conversations with people to determine prob- 
lems and issues does take time but does not 
really cost anything other than time. Basically 
it helps to monitor what is out there. There is 
usually quite a lot of secondary data available, 
whether from campus police, community 
police, judicial officers, or residence halls. At 
the University of Arizona we have looked not 
only at residence hall incidents but also things 
like damage or vandalism. While not all dam- 
age is alcohol related, a good portion of it is. 
Other data can come from student health cen- 
ters regarding alcohol-related incidents that 
result in students coming in for services. 

Some environmental factors that can be 
measured are how bars operate or advertise on 
campus. Monitoring a student newspaper or 
places where ads are posted on campus to 
determine the messages (frequency and con- 
tent) can be extremely important. An environ- 
mental scan can provide a broad cross section 
of the campus environment and is really not 
that difficult to do. Program staff or students 
can do this kind of environmental scan easily 
because they are in tune to issues on campus 
whereas an outside evaluator may not be. 

At the University of Arizona we have had stu- 
dents as well as community members involved 
with environmental scans. It has helped our 
community coalition members get engaged in 
the process. They have come on campus as well 
as gone into bars around campus to do envi- 
ronmental scans. Seeing for themselves what 
the issues are was extremely helpful in getting 
buy-in from the coalition members. 

Q: In times of limited resources for prevention 
on campus are there cost effective evaluation 
methods that can be implemented? 

A: In addition to environmental scans by stu- 
dents, program staff, and coalition members, 
there are other cost effective ways to do such 
things as collect information on individual 
outcomes. For example, online surveys are 
becoming more cost effective. Online survey 



programs can help campuses get a lot of data 
for a relatively small amount of money. In 
addition, most campuses also have existing 
evaluation resources, such as departments in 
which professors teach research and statistics. 
They often have graduate students looking for 
research projects. Many campuses have some 
kind of institutional research office that moni- 
tors things like retention and recruitment. 

Such offices also get all types of data and are 
often willing to work with departments to col- 
lect other survey information. They are a really 
good resource because their whole function is to 
collect data on campus. An institutional research 
office is a fantastic resource that generally costs 
the program very little, if anything. 

Q: At minimum what do you recommend 
programs do to evaluate their efforts? 

A: Programs need both process and outcome 
evaluation. Process evaluation really means 
documenting what was done, when it was 
done, whom it was done on or for or with, and 
any changes over time in programming. That 
is needed to publish findings and to get fund- 
ing. It is also what is needed to tell the admin- 
istration what you are doing. 

But the other half — and critical piece — is 
outcome evaluation. Unfortunately, a lot of 
evaluation stops with the process. Programs 
may get head tallies and satisfaction surveys, 
but that does not demonstrate that a program 
had an effect on reducing problems. For that, 
outcomes are needed — ^both environmental 
and individual. The purpose of environmental 
management is to change the environment to 
influence the individual. It is not sufficient 
just to evaluate at the environmental level. But 
it is also not sufficient just to look at individ- 
ual outcomes. That is part of getting evalua- 
tion involved in program planning. What are 
the outcomes you can truly expect if this pro- 
gram is effective? What would you expect 
immediately? What would you expect either to 
grow as an improvement or at least maintain 



Welcome New 
Network Members 

Developed in 1987 by the U.S. Department 
of Education, the Network Addressing 
Collegiate Alcohol and Other Drug Issues 
(Network) is a voluntary membership 
organization whose member institutions 
agree to work toward a set of standards 
aimed at reducing alcohol and other drug 
(AOD) problems at colleges and universities. 

The Network welcomes new members 
from across the nation, representing all types 
of institutions of higher education, from 
community colleges to universities. A list of 
new members who have joined since the last 
Catalyst issue was published is available here . 

The Network develops collaborative AOD 
prevention efforts among colleges and uni- 
versities through electronic information 
exchange, printed materials, and sponsor- 
ship of national, regional, and state activi- 
ties and conferences. Each Network member 
has a campus contact who, as part of the 
constituency of the region, helps determine 
activities of the Network. 

As of October 2007, Network membership 
stood at 1,600 postsecondary institutions. 

To learn more about the Network and 
how your campus can become a member, 
visit the Network’s Web site . ■ 



as an improvement over the long term? How 
will those outcomes be measured? Whether it’s 
a pre- and post-survey that looks at change 
over the course of a program or an e-mail 
pretest and a delayed posttest, it doesn’t have to 
be a lengthy survey. It just needs to cover the 
anticipated outcomes. 

It doesn’t have to take a huge amount of 
money or effort to come up with those things. 
The timing is really in the planning process. If 
evaluation is included at the very beginning it 
makes the whole process much easier. ■ 
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It’s Only Logical 

and how will that change in behavior translate to 
decreased rates of sexual assault? 

To achieve a complete program model, pro- 
gram planners should first make a separate 
“chain” showing how each activity leads to 
program goal(s). Typically, you will notice that 
as you move toward the right-hand side of each 
chain, similar results are expected. Planners 
can integrate the separate models into one 
large model by showing how separate activities 
and their immediate effects eventually lead to 
common outcomes on the right-hand side of 
the model. 

Let’s say that your campus survey and inter- 
views with campus and community stakehold- 
ers reveal that many underage students are 
drinking at certain bars nearby the campus. By 
observing these bars, you determine that IDs 
are not consistently and carefully checked at 
the doors of these establishments. Among other 
activities, you decide to try to increase the 
extent and consistency of ID checking at these 
bars with the objective of increasing the num- 
ber of underage patrons who are refused entry. 
With support from members of your campus 
and community coalition, the bar owners and 
managers are persuaded to establish a policy 
and protocol for universal ID checking and 
allow you to provide training to the bouncers. 
Bar owners are willing to support this policy 
because it helps them maintain their state 
liquor licenses in this state. 

Your logic model for the bouncer training 
“chain” and the policy change activity might 
look like the diagram at right. 

As you can see from the model, we expect 
bouncer training to result in a greater number 
of trained bouncers. Trained bouncers are 
expected to show increased knowledge about 
the new protocol and how to recognize fake 
IDs, an increase in positive attitudes toward 
checking IDs, and an increase in the skills 
required to refuse entry to a patron without 
proper ID (of course, the training must include 
content that addresses each of these goals). The 
changes in knowledge, attitudes, and skills are 



expected to lead to an increase in actual ID 
checking behavior according to the established 
protocol, which is assumed to lead to an 
increase in bouncers refusing service to under- 
age patrons. Serving fewer underage patrons 
is assumed to decrease the availability of 
alcohol to underage students, which is 
expected to decrease high-risk alcohol use and 
related consequences. The top row shows the 
policy change, protocol development, and 
ongoing management support for the policy. 
This part of the program could be represented 
in even more detail but serves to remind us 
that these factors are necessary to support the 
training efforts and subsequent behavior 
change by the bouncers. 

Notice that all of the relationships are articu- 
lated in terms of their “expected” or “assumed” 
results, which is the purpose of the logic 
model. By measuring each of these steps in the 
chain during the evaluation, we can deter- 
mine whether each of our assumptions is sup- 
ported, thereby informing future planning. 

Before implementing your activities, take a 
step back and consider whether your model is 
realistic and feasible. First, examine the 
changes specified in your model, and make 



sure that there is evidence from prior research 
or theory to support your assumptions. For 
example, many studies have shown that 
increasing students’ knowledge about the con- 
sequences of drinking will not result in a 
change in drinking behavior. Thus, if your 
model shows that you are expecting informa- 
tion about consequences to alter behavior, the 
research literature suggests that you should 
revise or replace that activity. Conversely, 
research shows that lower alcohol availability 
is associated with decreased consumption and 
problems, which provides an empirical basis 
for these links in the chain. Second, assess 
whether the program model is feasible given 
the available time, money, and staff — are too 
many activities proposed? The model can help 
you decide whether to delay, omit, or change 
some activities. Finally, given that most preven- 
tion programs take many years to reach ulti- 
mate program goals, ask yourselves whether 
you have specified intermediate goals that are 
realistic, measurable, and meaningful given 
the program activities and stakeholders. 

Linda Langford, Sc.D., is an associate director 
of the Higher Education Center for Alcohol and 
Other Drug Abuse and Violence Prevention. ■ 
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